Convergence of Nearest Neighbor Pattern Classification with Selective Sampling

In the panoply of pattern classification techniques, few enjoy the intuitive appeal and simplicity of the nearest neighbor rule: given a set of samples in some domain space whose value under some function is known, estimate the function anywhere in the domain by giving the value of the nearest sample (relative to some metric). More generally, one may use the modal value of the m nearest samples, where m ≥ 1 is some fixed integer constant, although m = 1 is known to be admissible in the sense that there is no m > 1 that is asymptotically superior in terms of prediction error [2]. The nearest neighbor rule is a nonparametric technique; that is, it does not make any assumptions about the character of the underlying function (eg, linearity) and proceed to estimate parameters modulo this assumption (eg, slope and intercept). Furthermore, it is extremely general, requiring in principle only that the domain be a metric space. The classic paper on nearest neighbor pattern classification is due to Cover and Hart [2]; a textbook treatment appears in Duda et al. [4]. Both presentations adopt a probabilistic setting, demonstrating that if the samples are independent and identically-distributed (iid), the probability of error converges to no more than twice the optimal probability of error, the so-called Bayes risk. In a fully deterministic setting, since the Bayes risk is zero, this amounts to showing that the nearest neighbor rule with iid sampling converges to the true pattern. Cover [1] extends these results to the estimation problem. Obviously iid sampling is almost certain to produce samples that are superfluous in the sense that the prediction remains equally accurate even if these samples are removed.